Object amount calculation apparatus and object amount calculation method

ABSTRACT

An object amount calculation apparatus includes: a receiver configured to obtain a first three-dimensional model and a second three-dimensional model different from the first three-dimensional model, each of the first three-dimensional model and the second three-dimensional model representing a same space, each of the first three-dimensional model and the second three-dimensional model being constituted with regions having respective attributes; and a processor configured to: align the first three-dimensional model and the second three-dimensional model based on at least one attribute of the first three-dimensional model and the second three-dimensional model; calculate, for each of the attributes, a difference between the first three-dimensional model aligned and the second three-dimensional model aligned; and output (i) a total amount of differences corresponding to two or more attributes among the attributes and (ii) information on the two or more attributes.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation application of PCT International Application No.PCT/JP2020/006390 filed on Feb. 19, 2020, designating the United Statesof America, which is based on and claims priority of U.S. ProvisionalPatent Application No. 62/813,351 filed on Mar. 4, 2019. The entiredisclosures of the above-identified applications, including thespecifications, drawings and claims are incorporated herein by referencein their entirety.

FIELD

The present disclosure relates to an object amount calculation apparatusand an object amount calculation method.

BACKGROUND

There is a technique of measuring an amount of an object in apredetermined space, such as the volume or weight (see Patent Literature(PTL) 1, for example).

With the technique disclosed in PTL 1, an image of a scene of alandslide is compared with an image of the same site before occurrenceof the landslide, and these images are aligned to be used for generatinga three-dimensional model, and the amount of disaster at the scene ofthe landslide is calculated based on the three-dimensional modelgenerated.

CITATION LIST Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No.2002-328021

SUMMARY

An object amount calculation apparatus according to an aspect of thepresent disclosure includes: a receiver configured to obtain a firstthree-dimensional model and a second three-dimensional model differentfrom the first three-dimensional model, each of the firstthree-dimensional model and the second three-dimensional modelrepresenting a same space, each of the first three-dimensional model andthe second three-dimensional model being constituted with regions havingrespective attributes; and a processor configured to: align the firstthree-dimensional model and the second three-dimensional model based onat least one attribute of the first three-dimensional model and thesecond three-dimensional model; calculate, for each of the attributes, adifference between the first three-dimensional model aligned and thesecond three-dimensional model aligned; and output (i) a total amount ofdifferences corresponding to two or more attributes among the attributesand (ii) information on the two or more attributes.

An object amount calculation method according to an aspect of thepresent disclosure includes: obtaining a first three-dimensional modeland a second three-dimensional model different from the firstthree-dimensional model, each of the first three-dimensional model andthe second three-dimensional model representing a same space, each ofthe first three-dimensional model and the second three-dimensional modelbeing constituted with regions having respective attributes; aligningthe first three-dimensional model and the second three-dimensional modelbased on at least one attribute of the first three-dimensional model andthe second three-dimensional model; calculating, for each of theattributes, a difference between the first three-dimensional model andthe second three-dimensional model which are aligned in the aligning;and outputting (i) a total amount of differences corresponding to two ormore attributes among the attributes and (ii) information on the two ormore attributes.

Note that the present disclosure may be implemented as a program thatcauses a computer to perform steps included in the above object amountcalculation method. The present disclosure may also be implemented as acomputer-readable, non-transitory recording medium, such as a CD-ROM, onwhich the program is recoded. The present disclosure may also beimplemented as information, data, or a signal indicating the program.The program, information, data, and signal may be distributed via acommunication network such as the Internet.

BRIEF DESCRIPTION OF DRAWINGS

These and other advantages and features will become apparent from thefollowing description thereof taken in conjunction with the accompanyingDrawings, by way of non-limiting examples of embodiments disclosedherein.

FIG. 1 is a diagram for describing an overview of an earth amountmeasurement system according to an embodiment.

FIG. 2 is a diagram for describing an overview of attribute extractionprocessing performed by the earth amount measurement system according tothe embodiment.

FIG. 3 is a diagram for describing an overview of earth amountcalculation processing performed by the earth amount measurement systemaccording to the embodiment.

FIG. 4 is a block diagram illustrating a configuration of the earthamount measurement system according to the embodiment.

FIG. 5 is a block diagram illustrating a configuration of an imagingapparatus included in the earth amount measurement system according tothe embodiment.

FIG. 6 is a block diagram illustrating a configuration of a controlapparatus included in the earth amount measurement system according tothe embodiment.

FIG. 7 is a block diagram illustrating a configuration of an earthamount calculation apparatus included in the earth amount measurementsystem according to the embodiment.

FIG. 8 is a flowchart illustrating three-dimensional model alignmentprocessing performed by the earth amount measurement system according tothe embodiment.

FIG. 9 is a diagram for describing alignment of two three-dimensionalmodels performed by an alignment circuit.

FIG. 10 is a sequence diagram illustrating processing of the earthamount measurement system according to the embodiment.

FIG. 11 is a flowchart illustrating the earth amount calculationprocessing performed by the earth amount measurement system according tothe embodiment.

FIG. 12 is a diagram illustrating an example of an image presented by auser interface included in the earth amount measurement system accordingto the embodiment when two three-dimensional models are to be aligned.

FIG. 13 is a diagram illustrating an example of difference informationpresented by the user interface included in the earth amount measurementsystem according to the embodiment.

DESCRIPTION OF EMBODIMENTS (Overview of the Present Disclosure)

An object amount calculation apparatus according to an aspect of thepresent disclosure includes: an obtaining circuit that obtains a firstthree-dimensional model and a second three-dimensional model differentfrom the first three-dimensional model, each of the firstthree-dimensional model and the second three-dimensional modelrepresenting a same space which is a predetermined space, each of thefirst three-dimensional model and the second three-dimensional modelbeing constituted with regions having respective attributes; analignment circuit that aligns the first three-dimensional model and thesecond three-dimensional model based on an attribute of the firstthree-dimensional model and the second three-dimensional model; and acalculation circuit that calculates an amount of a difference betweenthe first three-dimensional model and the second three-dimensional modelwhich are aligned by the alignment circuit, and outputs an attribute ofthe difference and the amount of the difference.

There is a conventional method of, for example, measuring (calculating)an amount of earth that has flowed into a predetermined space as aresult of a landslide or the like at a scene of a landslide disaster,for instance. Laser measurement can calculate the amount of earth athigh accuracy, but the calculation by the laser measurement istime-consuming and costly. In view of this, an object amount calculationapparatus according to the present disclosure, for example, compares athree-dimensional model representing a predetermined space in which anobject, such as earth, whose amount is to be calculated is absent (afirst three-dimensional model) and a three-dimensional modelrepresenting the predetermined space in which the object is present (asecond three-dimensional model). With this, the difference between thefirst three-dimensional model and the second three-dimensional model canbe simply and easily calculated by appropriately aligning the firstthree-dimensional model and the second three-dimensional model.Accordingly, the object amount calculation method according to thepresent disclosure can reduce the processing time for calculating theamount of the object.

For example, the object amount calculation apparatus according to anaspect of the present disclosure further includes: a generation circuitthat generates at least one of the first three-dimensional model or thesecond three-dimensional model from a plurality of images eachrepresenting the predetermined space.

This makes it possible to calculate the amount of the difference using asimple configuration for generating an image, such as a camera.

For example, when the difference has a plurality of attributes differentfrom one another, the calculation circuit classifies the plurality ofattributes according to type, and outputs classified attributes to causea display apparatus to collectively display the plurality of attributeson a type-by-type basis.

With this, the types of objects included in the difference areclassified and displayed by the display apparatus, and thus the user whochecks the display apparatus can easily understand the types of theobjects included in the difference.

For example, the calculation circuit calculates an amount included inthe amount of the difference on the type-by-type basis.

With this, the amounts of objects included in the difference aredisplayed by the display apparatus on a type-by-type basis, and thus theuser who checks the display apparatus can easily understand the amountsincluded in the amount of the difference on a type-by-type basis.

For example, the amount of the difference is at least one of a volume ora weight of the difference.

With this, the object amount calculation apparatus according to thepresent disclosure can calculate at least one of the amount or thevolume of the difference.

For example, the alignment circuit extracts a partial model from thefirst three-dimensional model and a partial model from the secondthree-dimensional model which have a common attribute, aligns thepartial models by shifting the partial models to align coordinates ofcorresponding positions in the partial models, and subsequently alignsthe first three-dimensional model and the second three-dimensional modelby shifting the first three-dimensional model and the secondthree-dimensional model in accordance with a shift of the partialmodels.

With this, the object amount calculation apparatus according to thepresent disclosure can align the first three-dimensional model and thesecond three-dimensional model without time-consuming processing such asalignment of the three-dimensional models through comparison of theshapes thereof. This makes it possible to further reduce the processingtime of the object amount calculation apparatus according to the presentdisclosure.

An object amount calculation method according to an aspect of thepresent disclosure includes: obtaining a first three-dimensional modeland a second three-dimensional model different from the firstthree-dimensional model, each of the first three-dimensional model andthe second three-dimensional model representing a same space which is apredetermined space, each of the first three-dimensional model and thesecond three-dimensional model being constituted with regions havingrespective attributes; aligning the first three-dimensional model andthe second three-dimensional model based on an attribute of the firstthree-dimensional model and the second three-dimensional model; andcalculating an amount of a difference between the firstthree-dimensional model and the second three-dimensional model which arealigned in the aligning, and outputting an attribute of the differenceand the amount of the difference.

With this, the difference between the first three-dimensional model andthe second three-dimensional model can be simply and easily calculatedby appropriately aligning the first three-dimensional model and thesecond three-dimensional model. Accordingly, the object amountcalculation method according to the present disclosure can reduce theprocessing time for calculating an amount of an object.

For example, in the obtaining, the first three-dimensional modelrepresenting the predetermined space before inflow of earth and thesecond three-dimensional model representing the predetermined spaceafter the inflow of the earth are obtained, and in the calculating, anamount of the earth is calculated as the amount of the difference. Withthis, the amount of earth, i.e., the difference, can be simply andeasily calculated by appropriately aligning the three-dimensional modelsrepresenting the predetermined space before and after a disaster thathas caused earth to flow into the predetermined space. Accordingly, theobject amount calculation method according to the present disclosure canreduce the processing time for calculating the amount of the earth.

Hereinafter, each embodiment of the present disclosure is described indetail with reference to the drawings. Note that each embodimentdescribed below illustrates a specific example of the presentdisclosure. Therefore, numerical values, shapes, materials, elements,the arrangement and connection of the elements, steps, the processingorder of the steps, etc. described in each embodiment below are mereexamples, and are not intended to limit the present disclosure.

Note also that the drawings are represented schematically and are notnecessarily precise illustrations. In the drawings, essentially the sameelements are given the same reference signs, and overlappingdescriptions thereof are omitted or simplified.

Embodiment [Overview]

FIG. 1 is a diagram for describing an overview of an earth amountmeasurement system according to an embodiment.

Earth amount measurement system 1000 according to the embodiment (seeFIG. 4, for example) includes a plurality of cameras 100 that capturethe inside of predetermined three-dimensional space A1 (hereinafter,capturing an image is also referred to as imaging, and three-dimensionalspace A1 is also simply referred to as space A1). To give a specificexample, space A1 is a space above road 20 on which earth 21 hasaccumulated. To give another specific example, space A1 is a space abovea river in which earth 21, driftwood, or the like has accumulated. Earthamount measurement system 1000 is an example of the object amountcalculation apparatus according to the present disclosure.

The plurality of cameras 100 are disposed in different positions, andcapture a common three-dimensional space (space A1, for example). Thus,imaging target regions of the plurality of cameras 100 in space A1include regions that at least partially overlap. For example, theplurality of cameras 100 are disposed in different positions to surroundpart of space A1 that includes earth 21 on road 20. Also, the pluralityof cameras 100 are oriented differently. The imaging target regions ofthe plurality of cameras 100 at least partially overlap. The imagingtarget regions at least partially overlap so as to allow generation(reconstruction) of a three-dimensional model (point group data, forexample) of space A1 based on video data obtained by imaging.

Note that the imaging target regions may overlap among some of cameras100, or may overlap among all cameras 100.

The plurality of cameras 100 each may be a moving camera that capturesimages at different points in time. When cameras 100 are moving cameras,the number of cameras 100 is not particularly limited, so long as thereis at least one camera 100.

In the processing performed in the three-dimensional reconstruction(processing of generating a three-dimensional model of space A1), aplurality of videos (hereinafter, videos are also referred to as images)obtained from the plurality of cameras 100 in the above manner are usedto calculate (generate) three-dimensional model 22 representing spaceA1. The processing performed in the three-dimensional reconstruction isto, for example, calculate three-dimensional model 22 by makingassociations among a plurality of images obtained by the plurality ofcameras 100 and performing geometric calculation based on theassociation relationship obtained.

Here, a three-dimensional model is data representing a geometric shapeof a predetermined three-dimensional space (space A1 in the presentembodiment). A model that represents a three-dimensional shape is, forexample, a point group including a plurality of three-dimensionalpoints. The three-dimensional model may be represented by voxels, or maybe represented by a mesh.

Note that these ways of representing a three-dimensional model are mereexamples, and do not limit the ways of representing a three-dimensionalmodel.

Data based on which a three-dimensional model is generated need not bean image, and is not particularly limited. For example, data based onwhich a three-dimensional model is generated may be data obtained from ameasuring instrument such as a LiDAR.

A LiDAR is a sensor for measuring a distance to an object. By causing aLiDAR to scan (move and measure) a predetermined region, the position ofan object located in the predetermined region can be calculated.

This type of measuring instrument such as the LiDAR generates athree-dimensional model by emitting electromagnetic waves and obtainingreflected waves that are the emitted electromagnetic waves which havebeen reflected by a measurement target. Specifically, the measuringinstrument measures a time period from the emission of electromagneticwaves to return of the electromagnetic waves after being reflected bythe measurement target, and calculates the distance between themeasuring instrument and a point on the surface of the measurementtarget using the measured time period and the wavelength of theelectromagnetic waves.

For example, the measuring instrument emits electromagnetic waves inpredetermined radial directions from a reference point of the measuringinstrument. For example, the measuring instrument emits electromagneticwaves at intervals of a first angle around the horizontal direction, andemits electromagnetic waves at intervals of a second angle around thevertical direction. Accordingly, the measuring instrument can calculatethe three-dimensional coordinates of a plurality of points on themeasurement target by calculating a distance between the measuringinstrument and the measurement target in each direction around themeasuring instrument.

The measuring instrument emits, for example, laser light as theelectromagnetic waves. Alternatively, the measuring instrument emits,for example, millimeter waves as the electromagnetic waves to measure adistance between the measuring instrument and the measurement target.

FIG. 2 is a diagram for describing an overview of attribute extractionprocessing performed by earth amount measurement system 1000 accordingto the embodiment. Specifically, (a) in FIG. 2 illustrates an example ofan image generated by capturing space A1. Illustrated in (b) in FIG. 2is a three-dimensional model representing space A1. Illustrated in (c)in FIG. 2 is an enlarged view of pixels of the image in (a) in FIG. 2which have an attribute “tree”.

Earth amount measurement system 1000 extracts one or more attributesusing a plurality of items of image data obtained by a plurality ofcameras 100 capturing space A1 or a three-dimensional modelreconstructing space A1.

Here, an attribute is information indicating a result of recognition ofthe meaning that a particular region in an image or a three-dimensionalmodel has. The attribute is information indicating that an attribute ofan object in a given region in an image is “tree”, for example. Anotherexample of the attribute is information indicating that an attribute ofa given partial point group in a three-dimensional model represented bya point group is “tree”. That is to say, “tree”, “driftwood”, “earth”,“house”, and so on are examples of the attribute. As described, thethree-dimensional model is constituted with regions having respectiveattributes, for example. The region is not particularly limited, and maybe a predetermined range of the three-dimensional model, or may be athree-dimensional point included in the three-dimensional model. Theregion may be a partial model which will be described later. In otherwords, the three-dimensional model may be constituted with partialmodels, which will be described later, having respective attributes.

The attribute extraction processing is implemented by semanticsegmentation, for example. Semantic segmentation is processing of givinga meaningful label to a pixel or a smallest unit of thethree-dimensional model, using information on a peripheral pixel or aperipheral smallest unit of the three-dimensional model.

To be specific, as illustrated in (c) in FIG. 2, semantic segmentationis, for example, processing of giving a label “tree” to each pixel thathas captured a tree present in a predetermined three-dimensional space.Each pixel can be labeled by referring to information on peripheralpixels. The information on peripheral pixels may be an R (red) G (green)B (blue) value, may be a luminance value, may be a distribution of pixelvalues, or may be a gradient. With this, a partial model which is a setof point groups to which the attribute “tree” has been assigned can bedetermined as a partial model having the attribute “tree”.

The extraction of the attribute is performed by, for example, patternrecognition based on predetermined data indicating “tree” etc. A usermay operate user interface 400 (see FIG. 4), which will be describedlater, to make an association between the attribute and an image or anobject included in the three-dimensional model.

FIG. 3 is a diagram for describing an overview of earth amountcalculation processing performed by earth amount measurement system 1000according to the embodiment. Specifically, (a) in FIG. 3 illustrates athree-dimensional model representing space A1 before a disaster.Illustrated in (b) in FIG. 3 is a three-dimensional model representingspace A1 after the disaster. Illustrated in (c) in FIG. 3 is differenceinformation calculation processing performed by earth amount measurementsystem 1000. Illustrated in (d) in FIG. 3 is an example of differenceinformation that is output by earth amount measurement system 1000.

Earth amount measurement system 1000 extracts information on earth 21(difference information) by calculating a difference using apre-disaster three-dimensional model (hereinafter also referred to as afirst three-dimensional model) and a post-disaster three-dimensionalmodel (hereinafter also referred to as a second three-dimensionalmodel), each of which has reconstructed space A1.

Earth amount measurement system 1000 extracts a part of the twothree-dimensional models that shows a noticeable change before and afterthe disaster. The result of difference calculation is presented to theuser as the difference information.

Here, the difference information is, for example, a volume of athree-dimensional partial region in which the pre-disasterthree-dimensional model and the post-disaster three-dimensional model donot overlap or intersect in plane in a three-dimensional mesh model. Theamount of difference calculated in such a manner is, for example, anamount of earth accumulated in a predetermined three-dimensional spaceafter the disaster.

The difference information may include the attribute that is obtained bythe attribute extraction and is the attribute of the part of the twothree-dimensional models that shows a noticeable change before and afterthe disaster, e.g., information on a mixture including a plurality ofattributes different from one another, such as proportions of driftwood,rubble, etc., included in earth 21.

The mixture is, for example, a set of partial models having differentattributes. In the above example, the mixture is earth. Further, themixture (i.e., earth) is a three-dimensional model including a partialmodel having an attribute indicating “driftwood” (a first partial model)and a partial model having an attribute indicating “rubble” (a secondpartial model). The difference information, for example, includes one ormore attributes included in the information on the mixture, includes thevolume of the mixture as the amount of the difference, and includesinformation indicating a proportion of the first partial model and aproportion of the second partial model in the volume of the mixture.

The mixture may be classified according to type of the attribute.Assume, for example, that the mixture includes a partial model having anattribute indicating “driftwood” (a first partial model), a partialmodel having an attribute indicating “rubble” (a second partial model),a partial model having an attribute indicating “traffic sign” (a thirdpartial model), and a partial model having an attribute indicating“traffic light” (a fourth partial model). In this case, the mixture maybe classified into a mixture including the first and the second partialmodels of natural objects (a first mixture) and a mixture including thethird and the fourth partial models of man-made objects (a secondmixture). In this case, the difference information may includeinformation indicating a proportion of the first mixture and aproportion of the second mixture in the volume of the mixture.

As described above, the difference information includes, for example,differential information indicating an amount of difference between twothree-dimensional models and the attribute of the difference. The amountof the difference is, for example, at least one of the volume or theweight of the difference.

Note that what is described above is a mere example of the differenceinformation, and does not limit the difference information.

As for the scale of the three-dimensional models, one of the twothree-dimensional models, that is, either the pre-disasterthree-dimensional model or the post-disaster three-dimensional model,may be adjusted to a desired scale. For example, the scale of one of thepre-disaster three-dimensional model and the post-disasterthree-dimensional model may be adjusted to the scale of space A1. Also,the scale of one of the pre-disaster three-dimensional model and thepost-disaster three-dimensional model may be adjusted based on scaleinformation provided from the outside of earth amount measurement system1000. By the adjustment that allows the three-dimensional models to havethe same scale, the difference can be calculated between thepre-disaster three-dimensional model and the post-disasterthree-dimensional model having the same scale. This makes it possible tocalculate the amount of earth with the scale of space A1.

Note that earth amount measurement system 1000 is an example of theobject amount calculation apparatus according to the present disclosure.The above-described method of calculating the difference information maybe used for calculating an amount of a target other than earth, such asan amount of stock in a warehouse, or may be used for calculating, forexample, an amount of construction materials at the location ofconstruction work.

[Configuration] <Earth Amount Measurement System>

A configuration of earth amount measurement system 1000 is now describedin detail.

FIG. 4 is a block diagram illustrating a configuration of earth amountmeasurement system 1000 according to the embodiment.

Earth amount measurement system 1000 includes a plurality of imagingapparatuses 10 a to 10 n (for example, n imaging apparatuses fromimaging apparatus 10 a, imaging apparatus 10 b, . . . to imagingapparatus 10 n, where n is an integer of 2 or greater), controlapparatus 200, earth amount calculation apparatus 300, and userinterface 400. Imaging apparatuses 10 a to 10 n are communicativelyconnected with control apparatus 200. Earth amount calculation apparatus300 and user interface 400 are also communicatively connected withcontrol apparatus 200.

Note that imaging apparatuses 10 a to 10 n may be communicativelyconnected with control apparatus 200 in a wired manner or a wirelessmanner. Earth amount calculation apparatus 300 and user interface 400may also be communicatively connected with control apparatus 200 in awired manner or a wireless manner.

User interface 400 is a reception appliance for receiving a useroperation. User interface 400 is implemented by, for example, a displayapparatus, a touch screen, a track pad, a keyboard, a mouse, or otherkinds of controllers, which are connected to control apparatus 200, or acombination thereof. User interface 400 is an example of a displayapparatus for displaying the attribute and the differential informationcalculated by the object amount calculation apparatus according to thepresent disclosure.

<Imaging Apparatuses>

FIG. 5 is a block diagram illustrating a configuration of imagingapparatus 10 a included in earth amount measurement system 1000according to the embodiment. Note that, hereinafter, since imagingapparatuses 10 a to 10 n illustrated in FIG. 4 have the sameconfiguration in the present embodiment, description focuses on imagingapparatus 10 a when, of the plurality of imaging apparatuses 10 a to 10n included in earth amount measurement system 1000, one imagingapparatus is to be described. Thus, the following description of imagingapparatus 10 a also applies to other imaging apparatuses 10 b to 10 n.

Imaging apparatuses 10 a to 10 n are apparatuses each including a cameraequivalent to camera 100 illustrated in FIG. 1, for capturing space A1.Possible configurations of imaging apparatuses 10 a to 10 n are thesame, for example.

Here, space A1 is a union of imaging regions of the plurality of cameras100.

Each of imaging apparatuses 10 a to 10 n includes camera 100, and mayinclude stand 110.

Camera 100 includes storage 101, controller 102, optical system 103, andimage sensor 104.

Storage 101 stores a program that is read and executed by controller102. Storage 101 temporarily stores video data on an imaging regioncaptured using image sensor 104, meta information such as a time stampattached to the video data, a camera parameter of camera 100, andimaging settings such as a frame rate or a resolution that is beingapplied.

Such storage 101 is implemented by use of a rewritable, nonvolatilesemiconductor memory such as a flash memory. In addition, a read-onlymemory (ROM), which is non-rewritable, or a random access memory (RAM),which is volatile, can also be used as storage 101 according to whetherdata to be stored needs to be overwritten, how long the data has to bestored, or the like.

Note that the number of imaging apparatuses included in earth amountmeasurement system 1000 is not particularly limited. For example, thenumber of imaging apparatuses included in earth amount measurementsystem 1000 may be two or more. In addition, imaging apparatuses 10 a to10 n included in earth amount measurement system 1000 need not havecommon properties. Also, cameras 100 included in imaging apparatuses 10a to 10 n are not limited to monaural cameras and may be stereo cameras.

Furthermore, the number of imaging apparatuses included in earth amountmeasurement system 1000 may be one if it is possible to capture space A1from a plurality of viewpoints by changing the position and theorientation of one imaging apparatus.

Controller 102 is implemented by use of a central processing unit (CPU),for example, and reads and executes the program stored in storage 101 tocontrol each element included in camera 100 so as to allow the imagingfunction and other functions to be carried out.

Note that controller 102 may be implemented by a dedicated circuit thatcontrols each element included in camera 100 to allow the imagingfunction and other functions to be carried out. In other words,controller 102 may be implemented by software or by hardware.

Optical system 103 is an element by which light from an imaging regionis formed into an image on image sensor 104, and is implemented by useof optical elements including a lens. The focal distance and the angleof view of optical system 103 may be changeable. A wide-angle lens or asuper-wide-angle lens such as a fisheye lens may be used in opticalsystem 103. For example, when videos captured by cameras included in theobject amount calculation apparatus according to the present disclosureare used in a monitoring system, wide-angle lenses may be used to expandan imaging region.

Image sensor 104 is implemented by a solid-state image sensor thatreceives light collected by optical system 103 with its light receivingsurface and converts the received light into an electric signalrepresenting an image, such as a charge-coupled-device (CCD) imagesensor, a complementary metal-oxide-semiconductor (CMOS) image sensor,or a metal-oxide-semiconductor (MOS) image sensor.

Stand 110 is an element that fixes and supports camera 100 in apredetermined position while camera 100 generates a video to be used forearth amount calculation by imaging, and is implemented by, for example,a tripod.

Note that the length and the angle of the leg(s) of stand 110 may beadjustable in order to adjust a fixing position of camera 100 aspreparation for the imaging.

Stand 110 may include a mechanism to rotate the pan head in order to panor tilt camera 100, an elevating mechanism to move camera 100vertically, and the like. Alternatively, stand 110 may include amechanism to support and move camera 100, such as a dolly or a crane.

Note that imaging apparatuses 10 a to 10 n need not necessarily befixed. Thus, imaging apparatuses 10 a to 10 n may have a configurationthat does not include stand 110.

Note that the number of imaging apparatuses included in earth amountmeasurement system 1000 is not particularly limited, so long as earthamount measurement system 1000 includes a plurality of fixed cameras,one or more moving cameras, or a combination thereof. Also, the camerasare not limited to monaural cameras, and compound-eye cameras such asstereo cameras may be included.

<Control Apparatus>

FIG. 6 is a block diagram illustrating a configuration of controlapparatus 200 included in earth amount measurement system 1000 accordingto the embodiment.

Control apparatus 200 includes storage 201, controller 202, and timer203.

Control apparatus 200 controls imaging apparatuses 10 a to 10 n, andreceives and outputs data from and to user interface 400. Also, controlapparatus 200 gives an earth amount calculation instruction to earthamount calculation apparatus 300 to perform earth amount calculation ondata received from imaging apparatuses 10 a to 10 n.

An example of control apparatus 200 is a computer. In this case, storage201 is a storage apparatus of the computer and is implemented by a harddisk drive, a semiconductor memory of any of various kinds, or acombination thereof. Controller 202 is implemented by a CPU of thecomputer.

Timer 203 is a timer included in control apparatus 200 and referred toby the CPU that implements controller 202.

Storage 201 stores a program that is read and executed by controller202. Storage 201 stores data that is received from imaging apparatuses10 a to 10 n and is to be processed by controller 202.

Controller 202 reads and executes the program stored in storage 201 soas to control above-described imaging apparatuses 10 a to 10 n and earthamount calculation apparatus 300.

Further, controller 202 performs processes in response to a userinstruction related to the above control and processing. One of theprocesses is the control on capturing synchronized videos by cameras 100included in imaging apparatuses 10 a to 10 n. Also, one of the processesmay include the earth amount calculation instruction.

Controller 202 functionally includes imaging controller 202 a, userinstruction obtainer 202 b, and earth amount calculation instructor 202c.

User instruction obtainer 202 b included in controller 202 is afunctional element that is implemented by controller 202 executing aprogram for obtaining a user instruction.

Further, earth amount calculation instructor 202 c included incontroller 202 is a functional element that is implemented by controller202 executing a program for the earth amount calculation instruction.

Note that imaging controller 202 a, user instruction obtainer 202 b, andearth amount calculation instructor 202 c of controller 202 may beimplemented by dedicated circuits that allow, for instance, imagingcontrol, user instruction obtainment, earth amount calculationinstruction, and earth amount calculation processing to be carried out.In other words, controller 202 may be implemented by software or byhardware.

Imaging controller 202 a causes imaging apparatuses 10 a to 10 n tocapture space A1 that is the imaging region, at different times. Imagingcontroller 202 a causes imaging apparatuses 10 a to 10 n to capture theimaging region in a state in which imaging apparatuses 10 a to 10 n arelocated in predetermined positions and oriented in predetermineddirections.

User instruction obtainer 202 b transmits, to user interface 400,imaging circumstance information provided from imaging apparatuses 10 ato 10 n, and obtains user input. The user input is a result of selectionof data targeted for the earth amount calculation, informationindicating whether the earth amount calculation processing is to beperformed, or a combination thereof. When the user input is informationindicating whether the earth amount calculation processing is to beperformed, user instruction obtainer 202 b outputs the informationindicating whether the earth amount calculation processing is to beperformed to, for example, earth amount calculation instructor 202 c.

Earth amount calculation instructor 202 c causes earth amountcalculation apparatus 300 to perform the earth amount calculationprocessing based on, for example, the information indicating whether theearth amount calculation processing is to be performed, which has beenreceived from user instruction obtainer 202 b. Further, earth amountcalculation instructor 202 c may cause earth amount calculationapparatus 300 to perform the earth amount calculation processing basedon the result of selection of data targeted for the earth amountcalculation. A specific example of the processing performed by earthamount calculation instructor 202 c will be described later.

<Earth Amount Calculation Apparatus>

FIG. 7 is a block diagram illustrating a configuration of earth amountcalculation apparatus 300 included in earth amount measurement system1000 according to the embodiment.

Earth amount calculation apparatus 300 includes storage 301 andprocessing circuit 302.

Earth amount calculation apparatus 300 processes data received viacontrol apparatus 200. Specifically, earth amount calculation apparatus300 performs the earth amount calculation processing on earth 21 that ispresent in space A1 captured by cameras 100 included in imagingapparatuses 10 a to 10 n.

The target of the earth amount calculation processing by earth amountcalculation apparatus 300 is, for example, space A1 after a disaster.With this, the amount of earth 21 (earth amount) accumulated in space A1can be determined by calculating the difference from thethree-dimensional model of space A1 before the disaster.

The target of the earth amount calculation processing by earth amountcalculation apparatus 300 need not be space A1 immediately after adisaster, and may be, for example, space A1 during earth removal work.With this, earth amount calculation apparatus 300 can calculate theprogress of the earth removal work by sequentially calculating theamount of earth.

The target of the earth amount calculation processing by earth amountcalculation apparatus 300 is not particularly limited, and need not bespace A1 before or after a disaster.

An example of earth amount calculation apparatus 300 is a computer. Inthis case, storage 301 is a storage apparatus of the computer and isimplemented by a hard disk drive, a semiconductor memory of any ofvarious kinds, or a combination thereof. Processing circuit 302 isimplemented by a CPU of the computer. Note that earth amount calculationapparatus 300 may be implemented by the same computer as that of controlapparatus 200.

Storage 301 stores a program that is read and executed by processingcircuit 302. Storage 301 also stores data received from imagingapparatuses 10 a to 10 n via control apparatus 200 and data on athree-dimensional model that is obtained from an external apparatus andis to be processed by processing circuit 302. That is to say, storage301 may store a three-dimensional reconstruction result, and may storean attribute extraction result. A pre-disaster three-dimensional modelmay be stored or a post-disaster three-dimensional model may be storedas the three-dimensional reconstruction result.

Processing circuit 302 reads and executes the program stored in storage301 so as to process the data received from imaging apparatuses 10 a to10 n. One of the processes is three-dimensional reconstruction of spaceA1.

Processing circuit 302 includes image obtaining circuit 302 a,three-dimensional reconstruction circuit 302 b, attribute extractioncircuit 302 c, model obtaining circuit 302 d, alignment circuit 302 e,and difference calculation circuit 302 f.

Image obtaining circuit 302 a obtains a plurality of images captured byimaging apparatuses 10 a to 10 n. Image obtaining circuit 302 a furtherobtains, along with the images, camera labels corresponding to theimages and indicating cameras 100 that have captured the images. Forexample, image obtaining circuit 302 a may obtain a plurality of imagesand camera labels by obtaining images to which camera labels have beenassigned. Each of the images may be a still image, or may be a movingimage.

Note that a plurality of images used for calibration processing(three-dimensional reconstruction processing) may be images captured bythe plurality of imaging apparatuses 10 a to 10 n at one correspondingtime (i.e., at the same point in time). The plurality of images obtainedby image obtaining circuit 302 a are stored in storage 301. Imageobtaining circuit 302 a may store the images and the camera labels instorage 301 in advance, prior to receiving the earth amount calculationinstruction from control apparatus 200. This allows earth amountcalculation apparatus 300 to start the earth amount calculationprocessing when the earth amount calculation is determined to benecessary.

Three-dimensional reconstruction circuit 302 b generates at least one ofthe first three-dimensional model or the second three-dimensional modelfrom the plurality of images each representing space A1. Morespecifically, three-dimensional reconstruction circuit 302 b calculatesa three-dimensional shape of space A1, using the plurality of imagesobtained by imaging apparatuses 10 a to 10 n capturing space A1. Forexample, three-dimensional reconstruction circuit 302 b generates athree-dimensional model of space A1 corresponding to a first point intime, using a group of images corresponding to the first point in time.Three-dimensional reconstruction circuit 302 b also generates athree-dimensional model of space A1 using a group of imagescorresponding to a point in time different from the first point in time.In such a manner, three-dimensional reconstruction circuit 302 bgenerates the first three-dimensional model and the secondthree-dimensional model that are of the same region but different inpoint in time.

Three-dimensional reconstruction circuit 302 b, for example, calculatesa three-dimensional shape of space A1 by making associations among theplurality of images captured by imaging apparatuses 10 a to 10 n andperforming geometric calculation based on the association relationship.The model representing a three-dimensional shape may be represented by apoint group, may be represented by voxels, or may be represented by amesh.

Note that these ways of representing a three-dimensional model are mereexamples. The ways of representing a three-dimensional model are notlimited to those mentioned above.

Attribute extraction circuit 302 c extracts an attribute for theplurality of images captured by imaging apparatuses 10 a to 10 n, thethree-dimensional model(s) reconstructed by three-dimensionalreconstruction circuit 302 b, or a three-dimensional model obtained bymodel obtaining circuit 302 d which will be described later. Forexample, attribute extraction circuit 302 c estimates, for athree-dimensional point group, an attribute of each point based oninformation on peripheral points, and calculates the estimated attributeas the attribute. Also, for example, attribute extraction circuit 302 cestimates, for image data, an attribute of an object captured in eachimage based on information on peripheral pixels, and associates theestimated attribute with each smallest unit of a three-dimensional modelreconstructed.

Attribute extraction circuit 302 c may obtain the three-dimensionalmodel(s) reconstructed by three-dimensional reconstruction circuit 302b, may obtain, from storage 301, the plurality of images captured byimaging apparatuses 10 a to 10 n, may obtain a three-dimensional modelobtained from storage 301, or may obtain a three-dimensional model frommodel obtaining circuit 302 d which will be described later.

Model obtaining circuit 302 d obtains a three-dimensional model fromthree-dimensional reconstruction circuit 302 b and storage 301. Forexample, model obtaining circuit 302 d obtains the model reconstructingspace A1 before the disaster from storage 301, and obtains the modelreconstructing space A1 after the disaster from three-dimensionalreconstruction circuit 302 b.

Note that model obtaining circuit 302 d may obtain two or morethree-dimensional models from storage 301. For example, model obtainingcircuit 302 d obtains the first three-dimensional model representingspace A1 and the second three-dimensional model representing space A1and different from the first three-dimensional model. Specifically,model obtaining circuit 302 d obtains the first three-dimensional modelthat represents space A1 at a first point in time and the secondthree-dimensional model that represents space A1 at a second point intime different from the first point in time. In the present embodiment,model obtaining circuit 302 d obtains two three-dimensional modelsbefore and after a disaster in space A1 from storage 301 (i.e., thefirst three-dimensional model representing space A1 before the disaster(at the first point in time) and the second three-dimensional modelrepresenting space A1 after the disaster (at the second point in time)).This make it possible to calculate the difference between the modelswithout three-dimensional reconstruction circuit 302 b performing thethree-dimensional reconstruction processing.

The three-dimensional model obtained by model obtaining circuit 302 dmay be a model calculated using image data outside earth amountmeasurement system 1000, or may be a model calculated by lasermeasurement outside earth amount measurement system 1000.

Note that the three-dimensional models obtained may be represented byvoxel data, may be represented by mesh data, or may be represented bypoint group data.

Using the two or more three-dimensional models obtained by modelobtaining circuit 302 d and the attribute calculated by attributeextraction circuit 302 c, alignment circuit 302 e performs processing ofaligning the coordinate systems of the three-dimensional models, thatis, processing of aligning the three-dimensional models. In other words,alignment circuit 302 e aligns the first three-dimensional model and thesecond three-dimensional model based on an attribute of the firstthree-dimensional model and the second three-dimensional model.Specifically, alignment circuit 302 e can efficiently compare the modelsby comparing, based on the attribute, parts which includethree-dimensional models having the same attribute and correspond topartial regions forming part of space A1.

While changing combinations within partial models having the sameattribute, alignment circuit 302 e makes an association among thepartial models.

Next, alignment circuit 302 e compares the smallest units betweenpartial models having one or more common attributes. For example,alignment circuit 302 e compares three-dimensional points in the casewhere the three-dimensional models are represented by point groups.

Here, a partial model is a set of the smallest units representing partor all of regions of a three-dimensional model.

Alignment circuit 302 e searches for closest points located closest toeach other between the partial models and makes an association betweenthe closest points, and adjusts the positions and the orientations ofthe coordinate systems to reduce the positional difference between eachpoint. Alignment circuit 302 e calculates a coordinate system thatminimizes the difference by repeatedly performing calculation whilechanging the partial models associated and the points associated, andperforms coordinate conversion on the three-dimensional model using thecoordinate system calculated.

As described above, alignment circuit 302 e performs alignment(coordinate conversion and size adjustment) based on an attribute. Forexample, alignment circuit 302 e extracts, from a plurality ofthree-dimensional models, three-dimensional models (partial models)having the same attribute among the plurality of three-dimensionalmodels. Alignment circuit 302 e compares the extracted three-dimensionalmodels to perform alignment. Further, alignment circuit 302 e changes anentire three-dimensional model in accordance with a partial modelchanged by the alignment. That is to say, alignment circuit 302 eextracts a partial model from the first three-dimensional model and apartial model from the second three-dimensional model which have acommon attribute, aligns the partial models by shifting the partialmodels to align coordinates of corresponding positions in the partialmodels, and subsequently aligns the first three-dimensional model andthe second three-dimensional model by shifting the firstthree-dimensional model and the second three-dimensional model inaccordance with the shift of the partial models.

This way, alignment circuit 302 e can more efficiently align thethree-dimensional models than when collectively aligning the entirethree-dimensional models.

The models resulting from the conversion may be stored in storage 301.

The partial models associated and the smallest units associated may bedetermined at random, or may be selected based on a feature amount.Further, the ways in which the partial models are associated and thesmallest units are associated are not limited to those described above.Note that when aligning colored models, information on color differencemay be used for the alignment.

As for the scale of the three-dimensional models, one of the twothree-dimensional models, that is, either the pre-disasterthree-dimensional model or the post-disaster three-dimensional model,may be adjusted to a desired scale. For example, it may be adjusted tothe scale of space A1. The scale of the three-dimensional model may beadjusted based on scale information provided from the outside of earthamount measurement system 1000. By the adjustment of the scale of thethree-dimensional model, the difference can be calculated between thepre-disaster three-dimensional model and the post-disasterthree-dimensional model having the same scale. This makes it possible tocalculate an amount of earth with the scale of space A1.

Difference calculation circuit 302 f calculates an amount of differencebetween the first three-dimensional model and the secondthree-dimensional model, and outputs an attribute of the difference anddifferential information indicating the amount of difference.Specifically, difference calculation circuit 302 f calculates differenceinformation indicating a difference between two or morethree-dimensional models whose coordinate systems have been aligned byalignment circuit 302 e. For example, difference calculation circuit 302f calculates the volume of a plane-covered three-dimensional region inwhich mesh models having the same coordinate system do not overlap orintersect in plane.

The difference information calculated by difference calculation circuit302 f may include an attribute of a region that shows a noticeabledifference, e.g., information on a mixture included in earth. Note thatthe above difference information is a mere example, and is not to limitthe present disclosure.

Difference calculation circuit 302 f outputs the calculated differenceinformation to user interface 400, for example. This allows the user tocheck the difference information (i.e., the attribute and the amount ofearth 21 that is the amount of the difference) by operating userinterface 400.

Further, for example, when the difference has a plurality of attributesdifferent from one another, difference calculation circuit 302 fclassifies the plurality of attributes according to type, and outputsthe classified attributes to cause a display apparatus (for example,user interface 400) to collectively display the plurality attributes ona type-by-type basis. In this case, difference calculation circuit 302 fcalculates an amount included in the amount of the difference on atype-by-type basis, for example.

[Processing Procedure] <Alignment Processing>

FIG. 8 is a flowchart illustrating three-dimensional model alignmentprocessing performed by earth amount measurement system 1000 accordingto the embodiment. FIG. 9 is a diagram for describing alignment of twothree-dimensional models performed by alignment circuit 302 e.

Alignment circuit 302 e included in earth amount calculation apparatus300 according to the present embodiment first makes an associationbetween partial models that are included in two or morethree-dimensional models and have the same attribute, e.g., betweenpartial models labeled “house” (S31). That is to say, in this case, twothree-dimensional models include partial models having an attribute thatindicates “house”, for example.

For example, as illustrated in (a) in FIG. 9, assume that a differencebetween first three-dimensional model 500 and second three-dimensionalmodel 510 is to be calculated. In this case, alignment circuit 302 efirst extracts partial models having an attribute that is included inboth first three-dimensional model 500 and second three-dimensionalmodel 510. Here, assume that alignment circuit 302 e has extracted firstpartial model 501 included in first three-dimensional model 500 andhaving an attribute “house” and second partial model 511 included insecond three-dimensional model 510 and having an attribute “house”.Next, as illustrated in (b) in FIG. 9, alignment circuit 302 e makes anassociation between first partial model 501 and second partial model511. Specifically, alignment circuit 302 e extracts one or more featurepoints included in first partial model 501 and one or more featurepoints included in second partial model 511, and makes associationsbetween the extracted feature points of first partial model 501 and theextracted feature points of second partial model 511. The feature pointsextracted here may be any feature points. For example, feature pointsare corners included in the partial models. Alignment circuit 302 emakes an association between an extracted feature point of first partialmodel 501 and an extracted feature point of second partial model 511which are the same. For example, alignment circuit 302 e makes anassociation between extracted feature point 520 of first partial model501 and extracted feature point 521 of second partial model 511.

Referring to FIG. 8 again, after step S31, alignment circuit 302 ecompares the smallest units between partial models having one or morecommon attributes, e.g., compares three-dimensional points in the casewhere the three-dimensional models are represented by point groups, and,for each point of the partial models, searches for a most closelylocated point and makes an association between the closest points (S32).

Next, alignment circuit 302 e adjusts the position and orientation ofthe coordinate system of a partial model to reduce the positionaldifference between the points associated in step S32 (S33). For example,as illustrated in (c) in FIG. 9, alignment circuit 302 e changes atleast one of first partial model 501 or second partial model 511 tolocate the associated feature points at the same coordinates. Forexample, alignment circuit 302 e shifts at least one of first partialmodel 501 or second partial model 511 to locate associated featurepoints 520 and 521 at the same coordinates. Further, alignment circuit302 e changes the entire three-dimensional model in accordance with thechange of the partial model. By doing so, alignment circuit 302 e makesa change such that models in first three-dimensional model 500 andsecond three-dimensional model 510 that represent the same object arelocated in the same position with the same orientation.

Note that alignment circuit 302 e may adjust the scale of athree-dimensional model when adjusting the position and the orientation.The scale may be adjusted to one of two three-dimensional models, or thescales of both of two three-dimensional models may be adjusted based onscale information obtained from the outside.

Referring to FIG. 8 again, after step S33, alignment circuit 302 edetermines whether the change in position and orientation of the partialmodel (for example, at least one of first partial model 501 or secondpartial model 511 illustrated in FIG. 9), which has been made in stepS33, is small (S34). The processing proceeds to step S32 when the changebrought about by the position and orientation adjustment is large, andproceeds to step S35 when the change is small.

An amount of change used here as the reference may be freely determinedin advance. For example, alignment circuit 302 e subsequently performsstep S32 when the amount of change before and after the adjustment ofthe position and orientation of the partial model is greater than orequal to a reference amount of change that has been freely determined inadvance. On the other hand, for example, alignment circuit 302 esubsequently performs step S35 when the amount of change before andafter the adjustment of the position and orientation of the partialmodel is less than the reference amount of change. An amount of changeis, for example, an amount of shift of each point. In the case ofrotating an entire partial model about a reference point, the amount ofchange may be the rotation angle of the rotation.

Next, alignment circuit 302 e compares the entire three-dimensionalmodels based on the result of the position and orientation adjustment,and determines whether the overall difference between the entirethree-dimensional models is large (S35). The processing proceeds to stepS36 when the overall difference is large, and proceeds to step S37 whenthe difference is small.

An overall difference used here as the reference may be freelydetermined in advance. For example, alignment circuit 302 e subsequentlyperforms step S36 when the difference before and after the adjustment ofthe position and orientation of the entire three-dimensional model isgreater than or equal to a reference difference that has been freelydetermined in advance. On the other hand, for example, alignment circuit302 e subsequently performs step S37 when the difference before andafter the adjustment of the position and orientation of the entirethree-dimensional model is less than the reference difference. Here, thedifference is, for example, an amount of three-dimensional points, intwo three-dimensional models, whose coordinates do not match.

Next, alignment circuit 302 e determines whether all possiblecombinations have been compared in steps S31 and S32 (S36). Theprocessing proceeds to step S37 when the comparison has finished for allpossible combinations, and proceeds to step S31 when the comparison hasnot finished. Here, for instance, when the processing proceeds to stepS31, alignment circuit 302 e extracts, for example, partial models whichhave not yet been compared between first three-dimensional model 500 andsecond three-dimensional model 510, and makes an association between theextracted partial models.

Note that the steps and the processing order of the steps illustrated inFIG. 8 are mere examples, and are not particularly limited. For example,the processing may proceed to step S36 after step S33. Further, stepsS34 and S35 may be excluded.

Finally, alignment circuit 302 e outputs a coordinate system thatminimizes the overall difference and a three-dimensional model obtainedby the coordinate conversion (S37).

<Sequence Diagram>

FIG. 10 is a sequence diagram illustrating processing of earth amountmeasurement system 1000 according to the embodiment.

Earth amount measurement system 1000 starts the earth amount calculationprocessing when a user processing instruction is received.

Control apparatus 200 transmits an imaging instruction to imagingapparatuses 10 a to 10 n (only imaging apparatus 10 a is shown in FIG.10), and transmits, to user interface 400, as imaging information,captured videos received from imaging apparatuses 10 a to 10 n, and userinterface 400 presents the imaging information to the user (S41). Theimaging information is, for example, an image of space A1.

The user operates user interface 400 to check the imaging informationand determine, for example, whether imaging apparatuses 10 a to 10 nhave appropriately captured a desired space and whether the imaginginformation is sufficient to start processing, and gives a processinginstruction (S42). Specifically, the user operates user interface 400 tocause user interface 400 to transmit information indicating theprocessing instruction to control apparatus 200.

The user processing instruction transmitted via user interface 400 isinformation or a signal indicating whether to start the earth amountcalculation processing or continue imaging by imaging apparatuses 10 ato 10 n. Note that the user processing instruction may includeinformation identifying which imaging information is to be used for theearth amount calculation processing or information specifying a resultto be output.

Next, based on the information indicating the user processinginstruction received from user interface 400, control apparatus 200determines whether to start the earth amount calculation processing orcontinue imaging by imaging apparatuses 10 a to 10 n (S43). When controlapparatus 200 determines that the earth amount calculation processing isto be started (Yes in S43), control apparatus 200 transmits the earthamount calculation instruction and the captured videos to earth amountcalculation apparatus 300.

On the other hand, when control apparatus 200 determines to continueimaging by imaging apparatuses 10 a to 10 n (No in S43), controlapparatus 200 transmits, to imaging apparatuses 10 a to 10 n,information indicating an instruction to continue imaging, so as tocause imaging apparatuses 10 a to 10 n to continue imaging.

Next, when earth amount calculation apparatus 300 receives informationindicating the earth amount calculation instruction from controlapparatus 200, earth amount calculation apparatus 300 performs the earthamount calculation processing using the captured videos based on theinformation received (S44). Step S44 is the processing of, for example,steps S31 to S37 illustrated in FIG. 9. After completion of the earthamount calculation processing, earth amount calculation apparatus 300transmits information indicating a processing result to controlapparatus 200.

Next, based on the user processing instruction, control apparatus 200organizes the processing result received from earth amount calculationapparatus 300, and outputs (transmits) the difference information touser interface 400 (S45).

User interface 400 presents the difference information received fromcontrol apparatus 200 to the user (S46). A specific example of thedifference information presented to the user will be described later.

<Summary of Processing Procedure>

FIG. 11 is a flowchart illustrating the earth amount calculationprocessing performed by earth amount measurement system 1000 accordingto the embodiment.

In the earth amount calculation processing performed by earth amountcalculation apparatus 300 according to the present embodiment, first,image obtaining circuit 302 a obtains a plurality of images captured byimaging apparatuses 10 a to 10 n (S51).

Next, three-dimensional reconstruction circuit 302 b performs thethree-dimensional reconstruction processing using the plurality ofimages (S52). For example, three-dimensional reconstruction circuit 302b generates a first three-dimensional model representing space A1 andconstituted with regions having respective attributes, and a secondthree-dimensional model representing space A1, constituted with regionshaving respective attributes, and different from the firstthree-dimensional model (for example, different in point in time).

Next, model obtaining circuit 302 d obtains two or morethree-dimensional models from three-dimensional reconstruction circuit302 b or storage 301 (S53). Note that model obtaining circuit 302 d mayobtain one or more three-dimensional models from three-dimensionalreconstruction circuit 302 b and one or more three-dimensional modelsfrom storage 301, or may obtain two or more three-dimensional modelsfrom storage 301.

Next, attribute extraction circuit 302 c extracts one or more attributesfrom the plurality of images captured by imaging apparatuses 10 a to 10n or from the three-dimensional models (S54). Note that step S54 may beperformed before step S53, may be performed after step S53, or may beperformed before and after step S53.

Next, alignment circuit 302 e performs processing of aligningcoordinates based on the two or more three-dimensional models and theattribute(s) (S55). Specifically, alignment circuit 302 e aligns thefirst three-dimensional model and the second three-dimensional modelbased on an attribute of the first three-dimensional model and thesecond three-dimensional model. For example, based on an attribute ofthe first three-dimensional model and the second three-dimensionalmodel, alignment circuit 302 e detects corresponding points that are thesame point in the first three-dimensional model and the secondthree-dimensional model, and aligns the first three-dimensional modeland the second three-dimensional model using the detected correspondingpoints. Alignment circuit 302 e can efficiently and accurately performthe alignment by making a comparison between regions having the sameattribute in the three-dimensional models (i.e., between partial modelshaving the same attribute).

Next, difference calculation circuit 302 f calculates the differenceinformation on a difference between the two or more models whosecoordinate systems have been aligned by alignment circuit 302 e (i.e.,the two or more models aligned by alignment circuit 302 e) (S56). Thedifference information includes, for example, differential informationindicating an amount of difference and the attribute of the difference.Difference calculation circuit 302 f, for example, outputs thecalculated difference information to user interface 400 (morespecifically, user interface 400 via control apparatus 200). Userinterface 400 presents (more specifically, displays) the obtaineddifference information to the user.

[Output Example]

FIG. 12 is a diagram illustrating an example of image 600 presented byuser interface 400 included in earth amount measurement system 1000according to the embodiment when two three-dimensional models are to bealigned.

User interface 400 is implemented by, for example, a mobile terminalsuch as a smartphone or a tablet terminal having a display apparatus andan input apparatus such as a touch panel display.

Note that the user may freely select the attribute to be used for thealignment, using a tablet-type mobile terminal as illustrated in FIG.12.

For example, image 600 includes three-dimensional image 601, targetselecting section 602, and determining section 603.

Three-dimensional image 601 is an image representing a post-disasterthree-dimensional model.

Target selecting section 602 is an image for receiving a selection of apartial model having an attribute which is to be preferentiallyassociated by alignment circuit 302 e. For example, of three-dimensionalimage 601 or target selecting section 602, the user touches an attributewhich the user wishes to be preferentially associated by alignmentcircuit 302 e or a partial model having the attribute. User interface400, for example, transmits information indicating the attribute or thepartial model having the attribute touched by the user to controlapparatus 200, for instance, after step S42 in FIG. 10, along with theinformation indicating the user processing instruction. Controlapparatus 200, for example, transmits the information indicating theattribute or the partial model having the attribute touched by the userto earth amount calculation apparatus 300 along with the earth amountcalculation instruction and the imaging information. Alignment circuit302 e, for example, in step S31, first selects a partial model based onthe information indicating the attribute or the partial model having theattribute touched by the user, and makes association.

FIG. 13 is a diagram illustrating an example of the differenceinformation presented by user interface 400 included in earth amountmeasurement system 1000 according to the embodiment.

User interface 400 displays, using a tablet-type mobile terminal asillustrated in FIG. 13, a result indicating an amount of earth which isan example of the calculated amount of difference. User interface 400may display, as the calculation result, an estimated weight of the earthother than the volume of the earth.

Further, user interface 400 may display a mixture of part correspondingto earth, using the attribute obtained by attribute extraction circuit302 c. This makes it possible to efficiently obtain informationnecessary for removing the earth and the mixture. The method ofdisplaying the mixture is not limited to characters; the mixture may bedisplayed by coloring the relevant part on the three-dimensional modelor the image.

For example, user interface 400 presents the difference information tothe user by displaying image 610. For example, image 610 includesthree-dimensional image 611, emphasis line 612, difference amountinformation image 613, and attribute image 614.

Three-dimensional image 611 is an image representing a post-disasterthree-dimensional model.

Emphasis line 612 is a line indicating, in three-dimensional image 611,a position, a region, etc. of the difference from a pre-disasterthree-dimensional model. Although FIG. 13 illustrates emphasis line 612with a circle drawn with a broken line for the purpose of description,emphasis line 612 may be any line such as a solid line, a broken line, abold line, or a colored line of a color different from the color of theother part, and may have any shape such as a quadrilateral shape or apolygonal shape.

Difference amount information image 613 is an image for presenting theamount of difference to the user. Difference amount information image613 includes, for example, information indicating the volume of thedifference and the estimated weight of the earth.

Attribute image 614 is an image for presenting the attribute of thepartial model included in the difference to the user. For example,attribute image 614 includes, as the attribute of the partial modelincluded in the difference, information indicating “earth”, “driftwood”,“traffic sign”, “traffic light”, etc. Further, in the present example,attribute image 614 includes the percentage of the volume or the weightof the part indicated by the attribute in the entire difference. In sucha manner, the amount of the difference may be displayed on anattribute-by-attribute basis.

In the present example, attribute image 614 includes first type image615 and second type image 616.

First type image 615 is an image for collectively presenting attributesthat satisfy a first condition among a plurality of attributes includedin the difference.

Second type image 616 is an image for collectively presenting attributesthat satisfy a second condition among the plurality of attributesincluded in the difference.

In such a manner, the user interface may collectively display, for eachtype of attribute, an attribute and an amount of the partial modelhaving the attribute.

Note that in the present example, the first condition is a naturalobject, and the second condition is a man-made object. These conditionsmay be freely determined in advance, and are not particularly limited.

[Advantageous Effects, Etc.]

As described above, an object amount calculation apparatus (for example,earth amount measurement system 1000; more specifically, earth amountcalculation apparatus 300) according to an aspect of the presentdisclosure includes: an obtaining circuit (for example, model obtainingcircuit 302 d) that obtains a first three-dimensional model and a secondthree-dimensional model different from the first three-dimensionalmodel, each of the first three-dimensional model and the secondthree-dimensional model representing a same space which is apredetermined space (for example, space A1), each of the firstthree-dimensional model and the second three-dimensional model beingconstituted with regions (for example, partial models orthree-dimensional points included in the three-dimensional models)having respective attributes; alignment circuit 302 e that aligns thefirst three-dimensional model and the second three-dimensional modelbased on an attribute of the first three-dimensional model and thesecond three-dimensional model; and a calculation circuit (for example,difference calculation circuit 302 f) that calculates an amount of adifference between the first three-dimensional model and the secondthree-dimensional model which are aligned by the alignment circuit, andoutputs an attribute of the difference and the amount of the difference.

There is a conventional method of, for example, calculating an amount ofearth that has flowed into a predetermined space as a result of alandslide or the like at a scene of a landslide disaster, for instance.Laser measurement can calculate the amount of earth at high accuracy,but the calculation by the laser measurement is time-consuming andcostly. In view of this, an object amount calculation apparatusaccording to the present disclosure, for example, compares athree-dimensional model representing a predetermined space in which anobject, such as earth, whose amount is to be calculated is absent (afirst three-dimensional model) and a three-dimensional modelrepresenting the predetermined space in which the object is present (asecond three-dimensional model). With this, the difference between thefirst three-dimensional model and the second three-dimensional model canbe simply and easily calculated by appropriately aligning the firstthree-dimensional model and the second three-dimensional model.Accordingly, the object amount calculation method according to thepresent disclosure can reduce the processing time for calculating theamount of the object.

For example, the object amount calculation apparatus according to thepresent disclosure further includes: a generation circuit (for example,three-dimensional reconstruction circuit 302 b) that generates at leastone of the first three-dimensional model or the second three-dimensionalmodel from a plurality of images each representing the predeterminedspace.

This makes it possible to calculate the amount of the difference using asimple configuration for generating an image, such as a camera.

For example, when the difference has a plurality of attributes differentfrom one another, the calculation circuit classifies the plurality ofattributes according to type, and outputs classified attributes to causea display apparatus (for example, user interface 400) to collectivelydisplay the plurality of attributes on a type-by-type basis.

With this, the types of objects included in the difference areclassified and displayed by the display apparatus, and thus the user whochecks the display apparatus can easily understand the types of theobjects included in the difference.

For example, the calculation circuit calculates an amount included inthe amount of the difference on the type-by-type basis.

With this, the amounts of objects included in the difference aredisplayed by the display apparatus on a type-by-type basis, and thus theuser who checks the display apparatus can easily understand the amountsincluded in the amount of the difference on a type-by-type basis.

For example, the amount of the difference is at least one of a volume ora weight of the difference.

With this, the object amount calculation apparatus according to thepresent disclosure can calculate at least one of the amount or thevolume of the difference.

For example, the alignment circuit extracts a partial model from thefirst three-dimensional model and a partial model from the secondthree-dimensional model which have a common attribute, aligns thepartial models by shifting the partial models to align coordinates ofcorresponding positions in the partial models, and subsequently alignsthe first three-dimensional model and the second three-dimensional modelby shifting the first three-dimensional model and the secondthree-dimensional model in accordance with a shift of the partialmodels.

With this, the object amount calculation apparatus according to thepresent disclosure can align the first three-dimensional model and thesecond three-dimensional model without time-consuming processing such asalignment of the three-dimensional models through comparison of theshapes thereof. This makes it possible to further reduce the processingtime of the object amount calculation apparatus according to the presentdisclosure.

An object amount calculation method according to an aspect of thepresent disclosure includes: obtaining (for example, step S53) a firstthree-dimensional model and a second three-dimensional model differentfrom the first three-dimensional model, each the first three-dimensionalmodel and the second three-dimensional model being three-dimensionalmodels representing a same space which is a predetermined space, each ofthe first three-dimensional model and the second three-dimensional modelbeing constituted with regions having respective attributes, aligning(for example, step S55) the first three-dimensional model and the secondthree-dimensional model based on an attribute of the firstthree-dimensional model and the second three-dimensional model; andcalculating (for example, step S56) an amount of a difference betweenthe first three-dimensional model and the second three-dimensionalmodel, and outputting an attribute of the difference and the amount ofthe difference.

With this, the difference between the first three-dimensional model andthe second three-dimensional model can be simply and easily calculatedby appropriately aligning the first three-dimensional model and thesecond three-dimensional model. Accordingly, the object amountcalculation method according to the present disclosure can reduce theprocessing time for calculating an amount of an object.

For example, in the obtaining, the first three-dimensional modelrepresenting the predetermined space before inflow of earth and thesecond three-dimensional model representing the predetermined spaceafter the inflow of the earth are obtained, and in the calculating, anamount of the earth is calculated as the amount of the difference.

With this, the amount of earth, i.e., the difference, can be simply andeasily calculated by appropriately aligning the three-dimensional modelsrepresenting the predetermined space before and after a disaster thathas caused earth to flow into the predetermined space. Accordingly, theobject amount calculation method according to the present disclosure canreduce the processing time for calculating the amount of the earth.

Other Embodiments

Although an object amount calculation apparatus and the like accordingto the present disclosure have been described based on the aboveembodiment, the present disclosure is not limited to the aboveembodiment.

For example, the above embodiment has described that the processingcircuit included in the object amount calculation apparatus and the likeis implemented by a CPU and a control program. For example, elements ofthe processing circuit may each be configured with one or more electriccircuits. Each of one or more electric circuits may be a general-purposecircuit, or may be a dedicated circuit. One or more electric circuitsmay include, for example, a semiconductor device, an integrated circuit(IC), a large scale integration (LSI) circuit, or the like. The IC orthe LSI circuit may be integrated into a single chip or may beintegrated into a plurality of chips. Although the name used here is ICor LSI circuit, it may be also called a system LSI circuit, a very largescale integration (VLSI) circuit, or an ultra large scale integration(ULSI) circuit, depending on the degree of integration. Further, afield-programmable gate array (FPGA) which is programmed after an LSIcircuit is fabricated may be used for the same purpose.

In addition, general or specific aspects of the present disclosure maybe implemented by a system, an apparatus, a method, an integratedcircuit, or a computer program. Alternatively, the general or specificaspects may be implemented by a computer-readable, non-transitoryrecording medium such as an optical disc, a hard disk drive (HDD), or asemiconductor memory in which the computer program is stored. Thegeneral or specific aspects may also be implemented by any combinationof systems, apparatuses, methods, integrated circuits, computerprograms, or recording media.

The division of the functional blocks in the block diagrams is a mereexample. A plurality of functional blocks may be implemented as a singlefunctional block, a single functional block may be divided into aplurality of functional blocks, and part of the functions may betransferred to another functional block. In addition, functions of aplurality of functional blocks having similar functions may be processedin parallel or by time-division by a single hardware or softwareproduct.

Furthermore, since the processing orders of the steps in the flowchartsare mere examples illustrated for specifically describing the presentdisclosure, other processing orders may be adopted. In addition, part ofthe steps may be performed simultaneously (in parallel) with anotherstep.

The present disclosure also encompasses other forms achieved by makingvarious modifications conceivable to those skilled in the art to theabove embodiment, as well as forms implemented by freely combining theelements and functions of the embodiment without departing from theessence of the present disclosure.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to an object amount calculationapparatus that calculates a difference from three-dimensional models,and is applicable to, for example, an apparatus and the like thatcalculate an amount of an object.

1. An object amount calculation apparatus, comprising: a receiverconfigured to obtain a first three-dimensional model and a secondthree-dimensional model different from the first three-dimensionalmodel, each of the first three-dimensional model and the secondthree-dimensional model representing a same space, each of the firstthree-dimensional model and the second three-dimensional model beingconstituted with regions having respective attributes; and a processorconfigured to: align the first three-dimensional model and the secondthree-dimensional model based on at least one attribute of the firstthree-dimensional model and the second three-dimensional model;calculate, for each of the attributes, a difference between the firstthree-dimensional model aligned and the second three-dimensional modelaligned; and output (i) a total amount of differences corresponding totwo or more attributes among the attributes and (ii) information on thetwo or more attributes.
 2. The object amount calculation apparatusaccording to claim 1, wherein the two or more attributes indicate amixture in the same space.
 3. The object amount calculation apparatusaccording to claim 1, wherein the two or more attributes indicate anatural object or a man-made object in the same space.
 4. The objectamount calculation apparatus according to claim 1, wherein the processoris further configured to: generate at least one of the firstthree-dimensional model or the second three-dimensional model from aplurality of images each representing the same space.
 5. The objectamount calculation apparatus according to claim 1, wherein the processoris configured to calculate an amount of the difference on anattribute-by-attribute basis.
 6. The object amount calculation apparatusaccording to claim 1, wherein an amount of the difference is at leastone of a volume or a weight of the difference.
 7. The object amountcalculation apparatus according to claim 1, wherein the processor isconfigured to extract a partial model from the first three-dimensionalmodel and a partial model from the second three-dimensional model whichhave a common attribute, align the partial models by shifting thepartial models to align coordinates of corresponding positions in thepartial models, and subsequently align the first three-dimensional modeland the second three-dimensional model by shifting the firstthree-dimensional model and the second three-dimensional model inaccordance with a shift of the partial models.
 8. The object amountcalculation apparatus according to claim 1, wherein the processor isconfigured to align the first three-dimensional model and the secondthree-dimensional model in a partial region which is a part of the samespace, based on a first partial attribute that the firstthree-dimensional model has for the partial region and a second partialattribute that the second three-dimensional model has for the partialregion, the second partial attribute being same as the first partialattribute.
 9. The object amount calculation apparatus according to claim8, wherein the processor is configured to shift at least one of thefirst three-dimensional model or the second three-dimensional model ofthe same space other than the partial region, in accordance with a shiftof at least one of the first three-dimensional model or the secondthree-dimensional model resulting from alignment of the firstthree-dimensional model and the second three-dimensional model in thepartial region.
 10. The object amount calculation apparatus according toclaim 1, wherein, when each of the first three-dimensional model and thesecond three-dimensional model has at least two attributes, theprocessor is configured to align the first three-dimensional model andthe second three-dimensional model based on a predetermined attributethat is included in the at least two attributes and is an attribute thateach of the first three-dimensional model and the secondthree-dimensional model has.
 11. The object amount calculation apparatusaccording to claim 10, further comprising: a user interface thatreceives an operation to give higher priority to the predeterminedattribute than to another attribute included in the at least twoattributes.
 12. An object amount calculation method, comprising:obtaining a first three-dimensional model and a second three-dimensionalmodel different from the first three-dimensional model, each of thefirst three-dimensional model and the second three-dimensional modelrepresenting a same space, each of the first three-dimensional model andthe second three-dimensional model being constituted with regions havingrespective attributes; aligning the first three-dimensional model andthe second three-dimensional model based on at least one attribute ofthe first three-dimensional model and the second three-dimensionalmodel; calculating, for each of the attributes, a difference between thefirst three-dimensional model and the second three-dimensional modelwhich are aligned in the aligning; and outputting (i) a total amount ofdifferences corresponding to two or more attributes among the attributesand (ii) information on the two or more attributes.
 13. The objectamount calculation method according to claim 12, wherein, in theobtaining, the first three-dimensional model representing the same spacebefore inflow of earth and the second three-dimensional modelrepresenting the same space after the inflow of the earth are obtained,and in the calculating, an amount of the earth is calculated as thetotal amount of the differences.
 14. A non-transitory computer-readablerecording medium for use in a computer, the recording medium having acomputer program recorded thereon for causing the computer to execute anobject amount calculation method including: obtaining a firstthree-dimensional model and a second three-dimensional model differentfrom the first three-dimensional model, each of the firstthree-dimensional model and the second three-dimensional modelrepresenting a same space, each of the first three-dimensional model andthe second three-dimensional model being constituted with regions havingrespective attributes; aligning the first three-dimensional model andthe second three-dimensional model based on at least one attribute ofthe first three-dimensional model and the second three-dimensionalmodel; calculating, for each of the attributes, a difference between thefirst three-dimensional model and the second three-dimensional modelwhich are aligned in the aligning; and outputting (i) a total amount ofdifferences corresponding to two or more attributes among the attributesand (ii) information on the two or more attributes.